Reference category prediction

Reference class forecasting is the outside-view method for resisting the planning fallacy: instead of predicting a project from its internal plan and the confidence of its planners, it begins with the observed outcomes of a broad class of comparable projects. Its decisive insight is that distributional information about what normally happens often predicts better than a detailed story about what planners hope will happen.

Origin in the Outside View

The method grows out of the distinction between the inside view and the outside view. The inside view concentrates on the particular project: its plans, tasks, obstacles, resources, and apparent reasons for expecting success. The outside view asks how similar projects have actually turned out. The distinction became concrete during a curriculum project organized after Daniel Kahneman persuaded officials in the Israeli Ministry of Education to support a high-school program on judgment and decision making. The team included experienced teachers, psychology students, and Seymour Fox, dean of the Hebrew University’s School of Education, and met every Friday afternoon for about a year while drafting the syllabus and textbook.

During that project, the participants confidentially estimated how long it would take to submit a finished textbook to the ministry. Their procedure was designed to elicit each person’s judgment before public discussion, thereby making better use of the knowledge distributed through the group. Yet the group’s estimate was unrealistically optimistic. When the members later confronted the outside view and the eventual outcome, they collectively ignored the statistical information. The episode illustrated why the method is needed: personal impressions of a vivid, specific case can overpower what broad evidence says about comparable cases.

Method and Procedure

Reference class forecasting begins by identifying an appropriate reference class, such as kitchen renovations or large railway projects. The class must contain projects sufficiently similar to the one being considered, while remaining broad enough to provide meaningful information about actual outcomes rather than merely repeating the current project’s assumptions.

The next step is to obtain the reference class’s statistics and use them to establish a baseline prediction. For transportation projects, the relevant database may contain plans and outcomes for hundreds of projects around the world. Such data can show typical cost per mile of railway, the percentage by which expenditures exceeded budget, the likely schedule overruns, and the degree to which projects of different types underperformed. The baseline is therefore anchored in what happened, not in the plausibility of the current plan.

Specific information about the individual project is not discarded. It is used afterward to adjust the baseline, but only when there are particular reasons to expect the optimistic bias or the likely outcome to differ from the reference class. This ordering matters: the statistical distribution is established first, and the case-specific narrative is allowed to modify it only on the basis of identifiable evidence. In this form, reference class forecasting combines the corrective force of the outside view with limited use of the inside view.

Correcting the Planning Fallacy

The planning fallacy describes plans and forecasts that lie unrealistically close to best-case scenarios. Initial budget errors are not always innocent. Planners may produce optimistic estimates because approval depends on them, while knowing that projects are rarely abandoned simply because costs or completion times exceed the original plan. Responsibility therefore also belongs to the decision makers who approve such plans: approving an estimate without demanding an outside view can institutionalize the planning fallacy.

The method’s central remedy is to prevent planners from underweighting or ignoring distributional information. The outside view does not depend on whether a project’s internal narrative feels coherent. It asks where the project sits within the actual distribution of comparable outcomes. For this reason, the method is especially valuable when a plan contains many individually reasonable steps but lacks a realistic account of the overruns, delays, and failures that commonly accumulate across the entire class.

Prediction, Expertise, and Calibration

The broader evidence on prediction explains why reference class forecasting is necessary even when a decision is entrusted to experts. Philip Tetlock’s twenty-year study collected more than 80,000 forecasts from 284 people who earned their living commenting on political and economic trends. The experts performed worse than they would have if they had assigned equal probabilities to three possible outcomes, and even in areas they knew best they were not significantly better than nonspecialists. More knowledge produced only a slight improvement, while the most knowledgeable forecasters were often less reliable because knowledge strengthened their illusion of skill and encouraged overconfidence.

A similar pattern appeared in a survey of 11,600 forecasts by chief financial officers of large corporations concerning the following year’s Standard & Poor’s index. The correlation between their estimates and the true market value was slightly less than zero, yet the forecasters did not seem to recognize that their predictions were worthless. Their confidence intervals were also far too narrow: outcomes fell outside the interval 67 percent of the time, although an 80 percent confidence interval should produce surprises only about 20 percent of the time.

Calibration provides a way to evaluate forecasts rather than trusting their tone. A properly calibrated judge’s stated percentile values should contain the corresponding proportion of actual outcomes; for example, 98 percent of true values should fall between the 1st and 99th percentile estimates. Studies instead found that actual values lay outside such extreme bounds about 30 percent of the time, revealing systematically excessive certainty. Reference class forecasting answers this problem by making the baseline depend on observed distributions rather than on the narrow confidence intervals generated by an individual’s current estimate.

Significance and Limits

The method belongs to a larger critique of intuitive prediction. Regression to the mean shows why extreme performances should not simply be repeated: an observed result may combine stable ability with unusually good or bad luck, so the next result is expected to move partly back toward average. The same logic applies to projects and other uncertain undertakings. An unusually smooth plan or an exceptionally favorable early estimate may contain temporary conditions that will not recur.

Reference class forecasting also exposes a tension between vivid evidence and statistical evidence. Individual cases readily feed System 1 and encourage stereotypes or persuasive narratives, whereas pallid statistical information is often neglected when it conflicts with personal impressions. The method’s significance lies not in eliminating judgment, but in disciplining it: first establish what comparable cases imply, then permit concrete differences to justify departures from that baseline. It converts forecasting from an act of storytelling into a structured comparison between a proposed case and the distribution of cases that came before it.